Context validation

Context validation is a concept introduced by Bill Dunn (1998, 2000). Context validity refers to the validity of inferences that we have estimated the proximal range of rival hypotheses. Context validation can be performed by a participatory bottom-up process to elicit from stakeholders rival hypotheses on causal relations underlying a problem and rival problem definitions. The function of context validation is to avoid so called type III errors. Type III error refers to assessing or solving the wrong problem by incorrectly accepting the false meta-hypothesis that there is no difference between the boundaries of a problem, as defined by the analyst, and the actual boundaries of that problem (Raifa, 1968, redefined by Dunn, 1997, 2000).

One could argue that such an open bottom up inclusion of stakeholder perspectives (be it rival hypotheses on a problem or rival constructs used to judge a problem) is endless as there are as many different perspectives as there are different citizens, and one does not know when one has covered the true range of perspectives in the stakeholder community. However, it has been shown that in projects where such elicitation processes were used, the cumulative distribution of unique rival hypothesis flattens out after a limited number of stakeholders, usually below 30 (Kloprogge et al., 2006).

Context validity refers to the probability that an estimate has approximated the true but unknown range of causally relevant constructs and hypotheses present in a particular policy context. Estimates of context validity have at least four requirements: character, correctness-in-the-limit, coordination, and cost effectiveness. 

Character
An estimate of the proximal range of rival hypotheses should have the same character as that which it estimates. Just as an estimate of a length should be a length, not a temperature, an estimate of the proximal range of plausible rival hypotheses should meet a character requirement. Plausible rival hypotheses should be subjectively meaningful to, and elicited from, stakeholders who support and oppose possible solutions to policy problems. Because knowledge of rival hypotheses is “socially embodied,” the search for rival hypotheses must occur in natural settings. In satisfying the character requirement, the process of observing rival hypotheses may occur by means of interviews, questionnaires, participant observation, ethnography, content analysis, or transcripts of public hearings or debates. But the character requirement cannot be made on the basis of quantitative data, or qualitative information, that is not based on (known) subjective meanings attached to rival hypotheses. Meeting the character requirement is necessary for appropriate estimates.

Correctness-in-the-Limit
As information on which an estimate is based becomes more complete, the estimate should eventually converge on the presumed true range of rival hypotheses in a natural knowledge system. A cumulative (rather than simple) frequency distribution of unique (nonduplicate) rival hypotheses may be arranged in order of decreasing frequency of occurrence and plotted on a line graph. The line graph should eventually flatten out, indicating that we have approached the proximal range of rival hypotheses. The correctness-in-the-limit requirement is necessary for unbiased estimates.

Coordination
An estimate of the proximal range of plausible rival hypotheses should coordinate with the shape of an asymmetric and positively skewed curve that characterizes the structure of words, concepts, ideas, and beliefs within knowledge systems of many kinds. Hypotheses supporting the beliefs of most stakeholders occur far more frequently than hypotheses opposing those beliefs. Rarely occurring hypotheses may be taken as symptoms of doubt, while those that occur more frequently are signs of trust in existing knowledge. A large ratio of trust to doubt, is a characteristic of knowledge systems of many kinds. Estimates are based on a system of hypotheses that are actually believed by stakeholders. The method of selecting stakeholders is “theoretical” sampling, which permits the selection of stakeholders who are not independent because they communicate with one another in natural knowledge systems. Theoretical sampling is more likely than random sampling to yield the coordinated distribution required for context validation. The coordination requirement is necessary for coordinated estimates.

Cost-Effectiveness
An estimate of the proximal range of rival hypotheses should be economical, as in “economy of cognition.” All knowledge systems are complex, and the total costs of an estimate vary with the relative complexity of a particular knowledge system. But evidence from areas as diverse as marketing, linguistics, bibliometrics, and psychotherapy suggests that the proximal range of a system of rival hypotheses is likely to be reached within a small number of probes. The “probative value” of a hypothesis, defined as its potential usefulness in challenging a knowledge claim, is inversely related to its frequency of occurrence. Frequently occurring hypotheses are less useful than rarely occurring hypotheses. The marginal probative value of an additional rival hypothesis increases at a diminishing rate, with zero probative value reached within a small number (less than 30) probes. The marginal cost of each new hypothesis increases at a constant rate. Thus, the achievement of approximate context validity is cost-effective, which signifies economy of cognition at the knowledge-system level. Costeffectiveness is necessary for efficient estimates. 

These four requirements of approximate context validity can be readily achieved in many circumstances. The fact that this achievement is even possible has practically significant implications for problem structuring in policy analysis and other applied sciences. To return to the parable of the drunk and the darkness, the drunk purchases economy of cognition at the expense of the scientific and practical benefits of testing rival hypotheses. Although looking under the lamppost appears preferable to searching in the dark, approximate context validity can be achieved by taking a guided expedition into what only appears as darkness. 

References
Dunn,W. N.: 1997, Cognitive Impairment and Social Problem Solving: Some Tests for Type III Errors in Policy Analysis, Graduate School of Public and International Affairs, University of Pittsburgh, Pittsburgh.

Dunn, W.N., Pragmatic eliminative induction: proximal range and context validation in applied social experimentation, GSPIA working paper 001, Graduate School of Public and International Affairs, University of Pittsburgh, 1998. 

Dunn, W.N., Using the method of context validation to mitigate type III errors in environmental policy analysis, GSPIA working paper 016, Graduate School of Public and International Affairs, University of Pittsburgh, 2000.

P. Kloprogge and J. van der Sluijs (2006), The inclusion of stakeholder knowledge and perspectives in integrated assessment of climate change. Climatic Change, 75 (3) 359-389.

Raifa, H.: 1968, Decision Analysis, Addison-Wesley, Reading, MA.